Real-time Delphi
   HOME

TheInfoList



OR:

Real-time Delphi (RTD) is an advanced form of the
Delphi method } The Delphi method or Delphi technique ( ; also known as Estimate-Talk-Estimate or ETE) is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The ...
. The advanced method "is a consultative process that uses
computer technology Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes, and development of both hardware and software. Computing has scientific, e ...
"Hartman, F. T., & Baldwin, A. (1995). Using Technology to Improve Delphi Method. Journal of Computing in Civil Engineering, 9(4), 244-249. to increase efficiency of the Delphi process.


Definition and idea

Gordon and PeaseGordon, T. J., & Pease, A. (2006). RT Delphi: An Efficient, “Round-less”, Almost Real Time Delphi Method. Journal of Technological Forecasting and Social Change,73(4), 321-333. define the advanced approach as an innovative way to conduct Delphi studies that do not involve sequential "rounds" and consequently lead to a higher degree of
efficiency Efficiency is the often measurable ability to avoid wasting materials, energy, efforts, money, and time in doing something or in producing a desired result. In a more general sense, it is the ability to do things well, successfully, and without ...
with regard to the time frame needed to perform such studies. Friedewald, von Oertzen, and CuhlsFriedewald, M., von Oertzen, J., & Cuhls, K. (2007). European Perspectives on the Information Society (EPIS) (Delphi Report Deliverable 2.3.1). European Techno-Economic Policy Support Network (ETEPS Net). underline that aspect by writing, in "a Real-Time-Delphi, the participants do not only judge twice but can change their opinion as often as they like when they see the aggregated results of the other participants". So, here it becomes clear that the Real-Time Delphi approach requires real-time
calculation A calculation is a deliberate mathematical process that transforms one or more inputs into one or more outputs or ''results''. The term is used in a variety of senses, from the very definite arithmetical calculation of using an algorithm, to th ...
and provision of group responses. Friedewald et al. further state that the Real-Time Delphi method has beneath its explorative and predictive elements also normative and communicative elements. These latter are investigated by Bolognini, who explores the potential of computer-based Delphi as a communication technique for
electronic democracy E-democracy (a combination of the words electronic and democracy), also known as digital democracy or Internet democracy, is the use of information and communication technology (ICT) in political and governance processes. The term is believ ...
. Comparative studies of von der Gracht and colleagues have revealed that Real-time Delphi studies are comparable to the outcome of conventional round-based Delphi surveys.


History

The basic idea of a real-time, therefore computer-based (usually
web-based A web application (or web app) is application software that is accessed using a web browser. Web applications are delivered on the World Wide Web to users with an active network connection. History In earlier computing models like client-serve ...
), Delphi approach originates in a paper published by Turoff back in 1972 about an online Delphi conference conducted in the
United States The United States of America (U.S.A. or USA), commonly known as the United States (U.S. or US) or America, is a country primarily located in North America. It consists of 50 states, a federal district, five major unincorporated territorie ...
.Turoff, M. (1972). Delphi Conferencing: Computer-Based Conferencing with Anonymity. Technological Forecasting and Social Change, 3, 159-204. The conference was characterized by remote locations of participants, an online tool to access and give judgments,
anonymity Anonymity describes situations where the acting person's identity is unknown. Some writers have argued that namelessness, though technically correct, does not capture what is more centrally at stake in contexts of anonymity. The important idea he ...
of the participants, continuous operations and analysis of results (i.e. participants were able to see given answers of the other participants in real-time), as well as asynchronous participation (i.e. participants could independently login and logout how often and when they desired). The stated aspects are some of the key characteristics of Real-Time Delphi studies, which shows that the original idea of conducting such studies can be traced back to the respective year. Today, nevertheless, technological
innovation Innovation is the practical implementation of ideas that result in the introduction of new goods or services or improvement in offering goods or services. ISO TC 279 in the standard ISO 56000:2020 defines innovation as "a new or changed entity ...
s and advanced computer aided design possibilities (e.g.
high-speed internet Internet access is the ability of individuals and organizations to connect to the Internet using computer terminals, computers, and other devices; and to access services such as email and the World Wide Web. Internet access is sold by Internet ...
connections, high definition graphic, and advanced processor performance) facilitate more sophisticated studies in this context.Häder, M. (2002). Delphi-Befragungen. Ein Arbeitsbuch. Wiesbaden: Westdeutscher Verlag. The general idea to develop a faster advanced form of Delphi studies by using ideas and basic concepts of Turoff, was initiated by the U.S.
Defense Advanced Research Projects Agency The Defense Advanced Research Projects Agency (DARPA) is a research and development agency of the United States Department of Defense responsible for the development of emerging technologies for use by the military. Originally known as the Adv ...
(DARPA), which awarded a grant in 2004 to develop an approach to improve "speed and efficiency of collecting judgments in tactical situations". A small software company named Articulate Software in
San Francisco San Francisco (; Spanish language, Spanish for "Francis of Assisi, Saint Francis"), officially the City and County of San Francisco, is the commercial, financial, and cultural center of Northern California. The city proper is the List of Ca ...
was awarded an innovation
research grant A grant is a fund given by an end entity grant – often a public body, charitable foundation, or a specialised grant-making institution – to an individual or another entity (usually, a non-profit organisation, sometimes a business or a loc ...
to develop what DARPA was asking for. Adam Pease, principal consultant and CEO of Articulate Software, published the findings and methodology together with Theodore Gordon in 2006. Based on the findings in this seminal paper, Heiko von der Gracht developed a second Real-time Delphi platform until 2008. In subsequent years, von der Gracht and colleagues used the platform extensively for prospective research studies. In 2011, von der Gracht and Gordon presented a joint paper at WFS Annual Summit 2011, which summarizes the lessons learned of 40 RTD studies across the two platforms. In 2017, Angenheyster and colleagues published a study, which compares various Real-time Delphi platforms. Analysis criteria include among others visual appearance, data output, user friendliness, administration, and types of questions. A cross-disciplinary study by Beiderbeck et al. focused on new directions and advancements of the Delphi method, including real-time formats. The authors provide a methodological toolbox for designing Delphi surveys including among others sentiment analyses of the field of psychology.


Differences between conventional and real-time Delphi method

The question arises how a Real-Time
Delphi study } The Delphi method or Delphi technique ( ; also known as Estimate-Talk-Estimate or ETE) is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The ...
differs from a Conventional Delphi study. The basic framework is to think of a Delphi study which is conducted in form of an online questionnaire. However, a Conventional round-based Delphi study conducted via the
internet The Internet (or internet) is the global system of interconnected computer networks that uses the Internet protocol suite (TCP/IP) to communicate between networks and devices. It is a '' network of networks'' that consists of private, pub ...
is called " Internet Delphi". The basic difference to Internet Delphi is that the process of a Real-Time Delphi is not characterised by single iterated rounds. In fact, real-time calculation and provision of responses are the key characteristics of Real-Time Delphis. Various other labels for Real-Time Delphi can be found in literature and many authors are not completely aware of the differences:Turoff, M., & Hiltz, S. (1995). Computer based Delphi processes. In M. Adler & E. Ziglio, E. (Eds.), Gazing into the Oracle: The Delphi Method and its Application to Social Policy and Public Health (pp. 56-88). London: Jessica Kingsley Publishers. "Electronic Delphi", "Computer Delphi", "Computer-aided Delphi", and "Technology Delphi". However, it is important to truly understand the design and process a researcher has chosen to find out whether real-time calculations and provisions have been applied or not. The typical Real-Time Delphi process can be described in the way that participants get access to an online questionnaire portal for a certain time frame, within which they are allowed to log in and log out as often as they want. Whenever they login, they will see all their
quantitative Quantitative may refer to: * Quantitative research, scientific investigation of quantitative properties * Quantitative analysis (disambiguation) * Quantitative verse, a metrical system in poetry * Statistics, also known as quantitative analysis ...
and qualitative answers of previous sessions and they can change all answers as desired within the given period of time. Besides their own answers they will see the ongoing – hence, real-time – responses of other participants, and with regard to
metric assessment Metric or metrical may refer to: * Metric system, an internationally adopted decimal system of measurement * An adjective indicating relation to measurement in general, or a noun describing a specific type of measurement Mathematics In mathem ...
s the group as a whole will be visualised in terms of median, average, and interquartile range (IQR). It has to be pointed out that the numerical visualisations as well as the qualitative inputs change in the course of other participants changing their responses. Consequently, a participant can find out to what extent his own responses from an earlier point of time are still within the group opinion (i.e. IQR). The core innovation, then, of Real-Time Delphi studies is the real-time calculation and provision of results.


Methodological advancements

The core methodological innovation of Real-Time Delphi studies are the absence of iterated rounds and the real-time calculation and provision of group responses. Whereas Conventional Delphi studies are characterised by repeating
sequential round In mathematics, a sequence is an enumerated collection of mathematical object, objects in which repetitions are allowed and order theory, order matters. Like a Set (mathematics), set, it contains Element (mathematics), members (also called ''eleme ...
s, the Real-Time Delphi approach is characterised by a continuous round-less procedure leading to a reduced time frame needed to conduct such studies.Gordon, T. J. (2007). Energy forecasts using a “Roundless” approach to running a Delphi study. Foresight, 9(2), 27-35. Consequently, conducting large-scale studies of huge complexity in a relatively short period of time becomes possible. Another core methodological innovation is the fact that experts may not only judge once or twice, depending on the number of rounds, as it has been usual in a Conventional Delphi study. During a Real-Time Delphi, experts can independently reassess their responses as often as they want. Hartman and Baldwin discuss further advantages of the Real-Time Delphi approach: First, the number of experts participating in the real-time study can be increased due to a higher degree of
automation Automation describes a wide range of technologies that reduce human intervention in processes, namely by predetermining decision criteria, subprocess relationships, and related actions, as well as embodying those predeterminations in machines ...
during and improved possibilities for analysis after the study. Additionally, the Internet provides the possibility to invite a worldwide expert panel to participate in the study. Second, the degree of interaction among the experts can be increased due to the fact that they can immediately react on others' comments. Additionally, the time frame between giving own answers and getting insights into others' responses is very short, which encourages stronger cognitive examination with the respective issue in question. Hartman and Baldwin argue that with the help of this procedure the validity of results is maximised. In order to conduct a Real-Time Delphi study,
computer software Software is a set of computer programs and associated documentation and data. This is in contrast to hardware, from which the system is built and which actually performs the work. At the lowest programming level, executable code consists ...
– usually web-based – is needed for facilitating real-time calculations and visualisation of results. It is generally proposed in the existing literature that the experts participating in the study see not only their own answers but also the median and interquartile range of all given responses immediately after answering a quantitative question. Besides the
quantitative assessment Quantitative research is a research strategy that focuses on quantifying the collection and analysis of data. It is formed from a deductive approach where emphasis is placed on the testing of theory, shaped by empiricist and positivist philoso ...
a qualitative judgment of participants can be shown which serves as a justification for their numerical assessment of the question. Additionally, it can be shown to the expert how many respondents have already given their answers. To examine the qualitative arguments of others participants can click on a button, and a "reasons window" opens, which shows the statements of other participants to underline their point of view. So, the respective legitimisations given by others may cause a respondent to recapture his own point of view. In the next step the expert can change his own answer, add new arguments to underline his point of view, or leave his answer unchanged. In addition, the
respondent {{unreferenced, date=February 2012 A respondent is a person who is called upon to issue a response to a communication made by another. The term is used in legal contexts, in survey methodology, and in psychological conditioning. Legal usage In ...
will be shown an attention indicator, a so-called "flag", if his answer is within or outside the
interquartile range In descriptive statistics, the interquartile range (IQR) is a measure of statistical dispersion, which is the spread of the data. The IQR may also be called the midspread, middle 50%, fourth spread, or H‑spread. It is defined as the difference ...
or significantly different from the median. This application helps to see and understand immediately the own assessment and to think about reasons for the deviation from group opinions or else a high degree of consent. The respondent's attention will be called by highlighting questions with a high degree of deviation with a different colour and by asking him to give further reasons for his deviation from the group opinion. After operating a question in the described procedure, the participant can continue to the next question or press a "save"-button in the program, which leads to an immediate update of the median, interquartile range, and given arguments, and then leave the program. A second advantage of the round-less approach is the fact that, in order to take part in the study, participants can login and logout with their personalised account as often as they want during the time frame provided. Their already given answers will be saved and recalled when they login the next time. So, by design of the study, there are no explicit single rounds to answer the questions. Updating and playing back the information to the other participants follow immediately in succession to the process of answering. Here, it becomes clear that the process of answering can be
synchronous Synchronization is the coordination of events to operate a system in unison. For example, the conductor of an orchestra keeps the orchestra synchronized or ''in time''. Systems that operate with all parts in synchrony are said to be synchronou ...
or
asynchronous Asynchrony is the state of not being in synchronization. Asynchrony or asynchronous may refer to: Electronics and computing * Asynchrony (computer programming), the occurrence of events independent of the main program flow, and ways to deal with ...
and a worldwide expert panel can be reached, which is one of the major advantages of web-based tools. Turoff and Hiltz argue that the issue of asynchronous interaction is probably one of the least understood characteristics of Real-Time Delphis. ZipfingerZipfinger, S. (2007). Computer-Aided Delphi: An Experimental Study of Comparing Round-Based with Real-Time Implementation of the Method. Linz: Trauner Verlag. points out two advantages of asynchronous participation of the experts: First, they can login to the portal whenever they want; therefore, one could argue that the degree of convenience of taking part is increased for the participant due to a 24-hour availability of the portal. Second, panellists can contribute to whatever aspects in the questionnaire they want, especially when having gone through each question at least once. Here, a substantial aspect of Real-Time Delphis becomes obvious: Turoff and Hiltz explain that a Real-Time Delphi study offers a design of structured communication which allows every individual to choose the sequence and speed to contribute to the problem solution process. So, in comparison to face-to-face discussions, the Real-Time Delphi approach gives room for individuality and different cognitive abilities of the participants. A further advantage is the fact that the administrator of the study can set an arbitrary time frame in which participants have to log in and take part in the questionnaire. So, whenever the
researcher Research is " creative and systematic work undertaken to increase the stock of knowledge". It involves the collection, organization and analysis of evidence to increase understanding of a topic, characterized by a particular attentiveness ...
or administrator of the study is satisfied with the existing answers (i.e. in terms of quantity and quality), he can declare the study to be ended and close the online tool (i.e. "freeze" the responses). The key features of Delphi studies, such as anonymity, controlled feedback and group response, are also met in the context of Real-Time Delphi studies. However, the issue of iteration is, by design of the technique, not valid for Real-Time Delphi studies anymore. Instead of answering each question a first time and getting a second sheet with the group responses in the second round, the Real-Time Delphi already shows the second screen (i.e. group responses) immediately after answering each question. Having answered each question at least once, the participant can usually control which question to reassess from a "consensus portal", which serves as a kind of control panel to access single questions again. So, on the one hand, the procedure differs from a Conventional Delphi and, on the other hand, the iteration into single rounds is missing. Having asked the question how the accomplishment of a Real-Time Delphi study differs from conducting a usual Delphi study, Gordon and Pease point out that a Real-Time Delphi study can be implemented via a site on the Internet or in any other network (e.g. intra-company network,
local area network A local area network (LAN) is a computer network that interconnects computers within a limited area such as a residence, school, laboratory, university campus or office building. By contrast, a wide area network (WAN) not only covers a larger ...
) and is, therefore, not conducted in paper-and-pencil form any more. As with all Delphi studies, the process of defining and selecting experts is still extremely important. The Conventional Delphi study is then divided into several steps of response round, analysis through the facilitator, playing back the information, next response round, and so on. However, the Real-Time Delphi study is, after granting access to the online tool, rather a self-running process. The basic strengths of a Real-Time Delphi study are its efficiency and applicability to all Delphi topics (i.e. common problem sets, decision making issues, cross impact studies, etc.). Figure 2 illustrates that the process of a Real-Time Delphi differs. Important is to point out that the number of interventions of the
facilitator A facilitator is a person who helps a group of people to work together better, understand their common objectives, and plan how to achieve these objectives, during meetings or discussions. In doing so, the facilitator remains "neutral", meaning t ...
needed during the response phases (i.e. after opening the online tool) are usually less. Having developed the online tool in advance, the intermediate analysis done by the facilitator of the study is rather uncomplicated in comparison to the Conventional Delphi. The overall shortened time period needed to conduct a real-time study underlines that the approach can be regarded as generally more efficient. Gordon and Pease point out that a Real-Time Delphi study is applicable for a wide range of possible circumstances under which the consultation of experts is necessary. On the one hand, the authors give the example of a "small group operating synchronously in a
conference room A conference hall, conference room, or meeting room is a room provided for singular events such as business conferences and meetings. Room It is commonly found at large hotels and convention centers though many other establishments, including even ...
with
laptop computer A laptop, laptop computer, or notebook computer is a small, portable personal computer (PC) with a screen and alphanumeric keyboard. Laptops typically have a clam shell form factor with the screen mounted on the inside of the upper li ...
s connected wirelessly to the web site where the software resides, with anticipated completion of the exercise in say 20 min.". On the other hand, it can be thought of a larger panel of experts operating asynchronously from remote locations within a longer period of time. The greatest weakness of the real-time approach is that it is missing a wholly integrated, scientifically founded concept. The real-time Delphi idea is still a very new concept, which requires further research and application to become a tool for full-scale operations. Especially the editing of the alpha (i.e. the first) inputs of respondents, the real-time presentation of group results, and the tracking of progress over time should be integrated in a kind of administrator package to make the accomplishment of a Real-Time Delphi less difficult.


Examples of real-time Delphi applications

Numerous examples for real-time Delphi applications can be found. Among them,
The Millennium Project The Millennium Project is a think tank that publishes a ''State of the Future'' report. In 2001 it was associated with the American Council for the United Nations University. ANDREW ZIPERN, November 8, 2001, The New York TimesRewarding Technolo ...
conducted by Glenn, Gordon, & Florescu in 2009 provides a context for global thinking and improved understanding of global issues, opportunities, challenges, and strategies. More information on the project can be found on www.millennium-project.org Another stream of projects based on real-time Delphi studies was conducted by the former Institute for Futures Studies and Knowledge Management of EBS University of Business and Law in
Germany Germany,, officially the Federal Republic of Germany, is a country in Central Europe. It is the second most populous country in Europe after Russia, and the most populous member state of the European Union. Germany is situated betwe ...
, among them the T&L2030 Series of five different studies together with
PricewaterhouseCoopers PricewaterhouseCoopers is an international professional services brand of firms, operating as partnerships under the PwC brand. It is the second-largest professional services network in the world and is considered one of the Big Four accounting ...

Transportation & Logistics 2030 Series
Further illustrative examples in academic journals include cases from logistics, social business, professional services, retail, and automotive.''Heading Towards a Multimodal City of the Future? Multi-Stakeholder Scenarios for Urban Mobility''. In: ''
Technological Forecasting & Social Change ''Technological Forecasting and Social Change'' (formerly ''Technological Forecasting'') is a peer-reviewed academic journal published by Elsevier which discusses futures studies, technology assessment, and technological forecasting. Articles focu ...
''. Volume 89, November 2014, Pages 201-221, doi:10.1016/j.techfore.2013.08.036.


Notes


References

* Detailed description of RTD platform and comparison of classical vs. RT Delphi: ''Validating an Innovative Real-Time Delphi Approach – A methodological comparison between real-time and conventional Delphi studies''. In: ''Technological Forecasting & Social Change''. 78, 9, 2011, p. 1681–1694, doi:10.1016/j.techfore.2011.04.006. * Comparison of different RTD platforms: ''Real-Time Delphi in practice — A comparative analysis of existing software-based tools''. In: ''Technological Forecasting & Social Change''. 118, 2017, p. 15-27, doi:10.1016/j.techfore.2017.01.023. * Early research in the field: ''Using the Delphi method to engage stakeholders: A comparison of two studies''. In: ''Evaluation and Program Planning''. 33, 2, 2010, p. 147-154, doi:10.1016/j.evalprogplan.2009.06.006. {{DEFAULTSORT:Real-Time Delphi Forecasting Systems thinking Estimation methods Futures techniques